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1  EXECUTIVE  SUMMARY 


Electronic  Systems  Center,  Hanscom  AFB  MA  in  conjunction  with  Lockheed  Martin 
Mission  Systems  conducted  a  two  level  Virtual  University  (VU)  proof  of  concept  to 
determine  the  effectiveness  of  delivering  Command  and  control  (C2)  operator  type-1 
training  electronically  using  synchronous  application  sharing  technology.  The  level  I 
trade  fly-off  invited  eight  vendors  to  participate  in  the  fly-off  to  complete  a  VU 
Requirements  Matrix.  The  down-select  process  was  based  on  the  results  of  evaluating 
responses  using  a  weighted  rating  methodology.  The  final  candidate  solutions  were 
evaluated  using  the  following  high-level  criteria:  product  integration,  requirements 
satisfaction,  product  stability,  and  SCORM  compatibility.  According  to  the  responses 
provided  on  the  requirements  matrix  only  two  vendors  met  all  the  requirements.  Level  II 
gave  each  of  the  two  vendors  the  opportunity  to  install  their  integrated  software  products 
and  run  tests  with  the  users.  After  more  than  three  weeks  of  testing  and  demonstrations  a 
joint  decision  was  made  to  cease  further  demonstrations  due  to  the  instability  of  one  of 
the  software  products.  Based  on  the  completed  Test  Cards,  Tester’s  Survey  (Usability 
and  Technology  Acceptance  Questionnaire),  Observer  responses,  instructor  evaluations 
and  assessments,  and  performance  data  VU  is  a  viable  solution  for  meeting  the  needs  of 
C2  Type-1  operator  training. 


2  INTRODUCTION 


The  Virtual  University  (VU)  is  described  as  a  computer-based,  on-line  environment 
enabling  real-time  interaction  between  students  and  instructors.  To  leverage  the  most 
suitable  software  for  this  project  LM  conducted  a  two-level  proof  of  concept  trade  study. 
For  Level  I  a  vendor  fly-off  was  conducted  to  solicit  potential  candidates  for  a  virtual 
classroom  solution  and  of  those  candidates  a  down-select  process  was  engaged  to 
determine  the  two  most  suitable  software  products.  For  Level  II  informal  and  formal 
demonstrations  were  conducted  using  a  participant  forum  consisting  of  instructors, 
students,  and  observers. 

In  Level  I  a  request  for  quote  was  provided  to  each  of  the  potential  vendor  candidates 
along  with  a  formalized  Requirements  Matrix.  The  Requirements  Matrix  was  based  on  a 
set  of  factors  considered  to  be  essential  to  the  function  of  a  VU.  These  factors  included 
the  ability  to  function  as  a  live  classroom,  be  employed  over  a  network  or  the  WWW 
using  existing  military  infrastructure,  enable  hands-on  use  of  TBMCS  software  and  the 
ability  to  operate  in  a  secure  internet  environment  (https).  Of  the  eight  vendor  recipients 
four  responded  with  a  completed  Requirements  Matrix  form.  The  four  respondents  were 
Centra,  Click21eam,  Intranet  U,  and  iLinc/NS  (Nelson  Stiltner)  Software.  A  weighted 
criteria  set,  based  on  the  TBMCS  Engineering  Notebook,  established  the  mechanism  to 
collect  responses  for  analysis  and  to  select  the  two  most  suitable  candidates. 

In  Level  II  the  two  selected  vendors  conducted  formal  and  informal  Government 
demonstrations  using  their  software.  For  demonstration  purposes  a  set  of  Test  Cards  and 


a  Tester’s  Survey  were  created.  The  Test  Cards  and  Tester’s  Survey  were  representative 
of  the  Theater  Battle  Management  (TBMCS)  VU  University  Proof  of  Concept  Test  Plan 
(ref.  12).  The  demonstrations  were  designed  and  conducted  in  a  manner  that  would 
provide  the  necessary  results  that  could  be  matched  to  the  items  listed  in  the 
Requirements  Matrix  and  categorized  as  shown  in  the  TBMCS  VU  Proof  of  Concept  Test 
Plan. 

During  Level  II  the  VU  design  demonstrated  the  use  of  existing  military  infrastructure  for 
delivery  to  joint  TBMCS  users.  The  design  demonstrated  an  integrated  product  with 
capability  of  both  synchronous  and  asynchronous  courses.  VU  demonstrated  the  use  of 
the  software,  pre/post  multiple  choice  testing,  the  ability  for  students  to  have  the 
capability  to  provide  a  critique  to  measure  the  students  self-assessment  of  his/her 
learning,  conduct  of  the  class,  and  progress  toward  their  capability  to  complete  the 
TBMCS  processes. 

Additionally,  as  part  of  the  Level  II  test  demonstrations,  performance  monitoring  data 
was  collected  assessing  total  bandwidth  usage/requirement,  web  page  response  time, 
audio  latency,  and  graphic  latency  on  shared  applications. 


2.1  REQUIREMENTS/CONTRAINTS 

The  project  shall  be  incrementally  developed  in  two  phases: 

•  Phase  I  will  consist  of  a  technology  proof-of-concept  demonstration  of  a  TBMCS 
VU.  Phase  I  will  determine: 

o  a)  if  joint  training  requirements  can  be  met  via  synchronous  distance 
learning  means, 

o  b)  if  the  current  DOD  infrastructure  can  support  synchronous  distance 
learning  technologies,  and 

o  c)  if  there  are  cost  and  time  savings  in  conducting  a  virtual  course  vs. 
mobile  training  teams. 

•  Phase  II  will  consist  of  converting  all  TBMCS  instructor  led  training  to  a  virtual 
environment 
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2,2  STATEMENT  OF  PROBLEM 


The  23  March  2003  TBMCS  VU  Project  Plan  (ref.  11)  stated  that  due  to  real  world 
issues,  a  strenuous  operational  tempo  has  been  established  among  the  services  that 
depletes  personnel  available  for  training.  Future  formal  classroom  training  for  service 
personnel  is  at  risk  due  to  less  instructor  availability  coupled  with  higher  costs.  This 
increases  the  challenge  to  meet  the  future  battlefield  training  community.  An  alternative 
method  for  sustaining  readiness  must  be  identified  and  implemented  in  order  to  preserve 
the  subject  matter  expert  knowledge  in  the  training  field. 


2.3  REFERENCES 

1.  Engineering  Notebook  (ENB)  for  the  Theater  Battle  Management  Core  Systems 
(TBMCS)  Integration  and  Development  Contract,  26  Apr  2002. 

2.  TBMCS  VU  Compliance — Centra,  26  Nov  03 

3.  TBMCS  VU  Compliance — lEinc,  26  Nov  03 

4.  TBMCS  VU  Implementation  Schedule,  12  Dec  03 

5.  TBMCS  VU  Performance  Test  Results,  23  Oct  03 

6.  TBMCS  VU  Performance  Test  Results,  05  Dec  03 

7.  TBMCS  VU  Trade  Study  Analysis  Centra,  27  Jun  03 

8.  TBMCS  VU  Trade  Study  Analysis  lEinc,  2  Jul  03 

9.  TBMCS  VU  Vendor  Customer  Survey,  24  Jul  03 

10.  TBMCS  VU  Weighted  Trade  Study  Analysis,  15  Jul  03 

11.  TBMCS  VU  Project  Plan,  23  Mar  2003. 

12.  TBMCS  VU  Proof  of  Concept  Test  Plan,  14  Nov  2003. 

13.  TTD03-16  VU  Request  for  Proposal,  04  Apr  2003. 

14.  TTD03-16  VU  System  Architecture,  25  Sept  2003. 

15.  TBMCS  VU  Trade  Study  Requirements  Matrix,  1 1  Sept  2003. 


3  LEVEL  I/VENDOR  TRADE  FLY-OFF 


3.1  LEVEL  I  OVERVIEW 

The  purpose  of  a  vendor  trade  fly-off  was  to  determine  whieh  vendor  produets  meet  the 
needs  for  the  VU.  A  series  of  events  took  plaee  during  the  trade  fly-off.  First,  the  Trade 
Study  Seleetion  Criteria  established  the  basie  playing  field  for  determining  whieh 
vendors  would  be  suitable  to  partieipate  in  VU.  Second,  the  VU  Requirements  Matrix 
was  created  based  on  the  high-level  selection  criteria.  The  Requirements  Matrix  (ref.  15) 
along  with  a  RFQ  was  submitted  to  the  identified  vendors.  Third,  based  on  the  results  of 
the  returned  Requirements  Matrix,  a  weighted  study  was  generated  to  select  the  two  most 
suitable  products.  Forth,  a  vendor/customer  survey  was  conducted  to  help  insure  the  two 
best  candidates  were  selected  for  the  VU  demonstrations. 


3.2  TRADE  STUDY  SELECTION  CRITERIA 

To  be  an  effective  tool,  the  VU  software  must  be  able  to  perform  the  classroom  tasks 
necessary  to  teach  TBMCS  operators  to  use  TBMCS  while  performing  within  the  WWW 
and  military  security  environment. 

3.2.1  PRODUCT  INTEGRATION 

•  Single  Software  with  all  components 

•  Electronic  Learning  Management  System  (ELMS) 

•  Collaborative  virtual  classroom 

•  Scheduling 

•  Survey  Program 

3.2.2  REQUIREMENTS  SATISFACTION 

•  Meets  requirements 

•  Special  requirements 

•  Security  (HTTPS) 

•  Platforms  supported  (Internet  Explorer  &  Netscape) 

3.2.3  PRODUCT  STABILITY 

•  Software  length  in  use 

•  Number  of  companies  using  software 

3.2.4  SCORM  COMPATIBILITY 

•  Software  meets  SCORM  standards 
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3.3  VENDOR  REQUIREMENTS  MATRIX 

The  following  set  of  VU  requirements  (ref  15)  was  used  to  eompare  the  eandidate 
solutions  in  this  study.  These  eriteria  established  the  basis  for  the  weighted  study.  The 
following  list  of  requirements  were  answered  yes  by  both  vendors  during  the  trade  fly- 
off.  validating  that  they  eould  meet  all  the  requirements.  The  following  requirements 
were  reviewed  before,  during,  and  after  the  demonstration  phase  of  eaeh  of  the  two 
vendors. 


3.3.1  SERVER  BASED 

The  VU  solution  eandidates  were  evaluated  based  on  the  ability  to  run  on  a 
Government  furnished  server,  on  the  LM  extra-network  eonneeted  to  the  WWW  and 
.mil  NIPERNet  with  elient  aeeess  through  HTTPS  Port  443. 


3.3.2  CONNECTION 

•  The  VU  solution  eandidates  were  evaluated  based  on  the  following 
eonneetion  eriteria: 

•  Minimum  eonneetion  speed  is  28.8  kbps 

•  Aeeess  through  Seeure  Soeket  Layer  (SSL) 

•  Aeeess  through  a  stateful  inspeetion  firewall 

•  Aeeess  using  proxy 

3.3.3  SYNCHRONOUS  CLASSES 

•  The  VU  solution  eandidates  were  evaluated  based  on  the  following 
Synehronous  eriteria; 

•  Application  Sharing  -  Single  User 

•  Application  Sharing  -  Multiple  Simultaneous  Users 

•  Application  Sharing  -  Cross  Platform 

3.3.4  COLLABORATION 

•  The  VU  solution  candidates  were  evaluated  on  the  following  collaboration 
criteria: 

•  Text  Chat  -  One  to  One 

•  Text  Chat  -  One  to  Many 

•  Text  Chat  -  Many  to  One 

•  Audio  -  Voice  over  IP  (VOIP) 

•  Audio  -  Conference  Call 

•  Electronic  Bulletin  Board 

•  Desktop  Video  session  Recording  &  Playback  with  audio  and  editing 

•  Breakout  Rooms 

•  Hand  Raising  (polling) 

•  Threaded  discussion 


3.3.5  TESTING  AND  ADMINISTRATION 

•  Integration  of  SCORM  Compliant  Testing  in  Hyper  Text  Markup 
Language  (HTML) 

•  Asynchronous  Administration 

•  Auto  grading  and  feedback  for  question  scoring  for  individual  question  or 
entire  test 

•  Auto  posting  of  test  scores  to  student  grade  book 

•  Student  remediation  on  test  questions 

•  Student  competency 

•  Test  question  analysis 

3.3.6  ASYNCHRONOUS  CLASSES 

•  Bookmark  capability 

•  Pre/Post  Test  Administration 

•  HTML/Extensible  Markup  Language  (XML)  interactive  courseware 
administration 

•  Create  and  administer  surveys 

3.3.7  SCORM  COMPLIANCE 

•  Navigation  and  sequencing  between  learning  resources 

•  Sequencing  of  lesson  module  SCO  into  courses 

3.3.8  ENROLLMENT  ADMINISTRATION 

•  Training  manger  enrolls  students  through  browser 

•  Student  can  self  enroll 

•  Enrollment  by  job  position  or  course 

•  Training  manager  and  student  can  view  and  edit  the  student’s  training  plan 

•  View  student  history  and  progress  toward  completion  of  training  plan 

3.3.9  COURSE  SCHEDULING 

•  Scheduling  a  single  class 

•  Scheduling  courses 

•  Training  Calendar 

3.3.10  STUDENT  TRACKING 

•  Lesson  start 

•  Lesson  completion 

•  Tests  pass/fail  scoring 

•  Objectives  definitions  for  certification 

•  Objectives  completed  for  certification 

•  Competency  definitions — student’s  progress 

•  Track  student  performance/progress 
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•  Unique  alphanumeric  passwords 

•  Uses  official  names  for  students  in  the  enrollment  roster 

•  Uses  a  ‘credit’  variable  to  indicate  a  pass/fail 

•  Uses  a  ‘lesson  status’  variable  to  indicate  a  student’s  status  in  a  lesson 

•  Uses  an  ‘entry’  variable  to  indicate  if  a  student  has  started  the  lesson  but 
never  completed 

3.3.11  REPORTS 

•  End-user  customizable 

•  Individual  student  reports 

•  Class  reports;  demographics,  total  attendance,  total  completion 

•  Export  delimited  file  to  Excel 

•  Test  question  analysis 

•  Graphic  presentation  of  analysis  data 

•  Cumulative  class  and  course  reports 

•  Performance  data  collection  and  reporting 

3.3.12  SURVEYS 

•  Pre-built  surveys 

•  Customizable  surveys 

•  Automated  compilation  of  summary  data,  and  statistical  graphing 

•  End-of-course  surveys 

•  Self-assessment  surveys 

•  Comments  survey 

•  Export  delimited  file  to  Excel 

3.3.13  COST 

•  Server  media 

•  Client  media  (25  students,  50  students,  or  75  students) 

•  Documentation 

•  Annual  maintenance 

•  Annual  technical  support 

•  Initial  setup  and  training 

3.4  WEIGHTED  STUDY 

A  weighted  study  was  accomplished  to  compare  the  four  vendor  responses  in  order  to 
analyze  and  measure  the  results  based  on  the  Requirements  Matrix  (ref  15).  The 
weighted  study  was  based  on  the  TBMCS  ENB  (ref  1).  Below  lists  the  categories  of  the 
study  and  the  total  score  received  for  each  of  the  four  vendors.  The  two  most  significant 
areas  that  caused  the  other  two  vendors  to  receive  lower  scores  were  not  having  a  fully 
integrated  product  (including  an  EMS)  and  the  ability  to  run  on  a  28.8  connection. 

•  Product  Integration 

•  Requirements  Satisfaction 


•  Product  Stability 

•  SCORM  Compatibility 

•  Future  Product  Migration  and  Implication 

•  Company  Stability 

•  Company  Technical  Support 

Weighted  Study  results  of  the  four  vendors: 

•  ILinc/NS  Software  308 

•  Centra  301 

•  Intranet  U  280 

•  Cliek21earn  259 

• 

To  further  review  the  results  of  the  weighted  study  refer  to  TBMCS  VU  Weighted  Trade 
Study  Analysis  (ref  10) 

3.5  VENDOR/CUSTOMER  SURVEY 

Prior  to  selecting  and  procuring  the  final  two  vendors  for  partieipation  in  the  product 
demonstrations  vendor  referenees  were  aequired  from  eaeh  of  the  four  vendors.  Upon 
eompletion  and  review  of  the  Weighted  Trade  results,  the  VU  team  eontaeted  eaeh  of  the 
four  vendors  for  a  list  of  eurrent  customers  having  similar  training  requirements  as  the 
TBMCS  VU  eoneeptual  environment.  The  survey  suggested  that  all  customers  were 
eompletely  satisfied  with  the  produets  and  the  customer  serviee.  To  further  review  the 
results  of  the  eustomer  survey  refer  to  TBMCS  VU  Vendor  Customer  Survey  (ref.  9) 


4  LEVEL  II/VENDOR  DEMONSTRATIONS 


4.1  LEVEL  II  OVERVIEW 

A  series  of  events  took  plaee  during  the  vendor  demonstrations.  First,  the  two  vendors 
Centra  and  iLinc/NS  Software  were  procured  and  sehedules  were  established  for 
teehnieal  personnel  and  Symposium  instructor  to  arrive  at  the  LM  facility.  Second, 
performance  monitoring  was  proeured  and  a  sehedule  was  established  for  teehnieal 
personnel  to  arrive  at  the  LM  faeility.  Third  the  Test  Plan,  Test  Cards,  and  Tester’s 
Survey  and  ereated  and  available  to  begin  testing  and  demonstration  of  the  vendor 
produets.  Fourth,  produet  testing,  informal  demonstrations,  formal  demonstrations  along 
with  performanee  monitoring  were  eondueted. 
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4,2  TEST  PLAN 


4.2.1  SUMMARY 

The  purpose  of  the  test  plan  was  to  define  a  meehanism  for  eapturing  the  neeessary 
data  and  eolleetion  of  results  needed  to  insure  a  competent  and  comprehensive 
decision  is  made.  The  test  plan  established  a  direct  correlation  between  the  testing 
objective  and  the  testing  results.  The  TBMCS  VU  Proof  of  Concept  Test  Plan  (ref 
12)  defines  four  separate  objectives  that  are  correlated  to  test  cards,  questionnaires,  as 
well  as  the  requirements  matrix  that  was  supplied  to  the  vendors  during  Level  I  of  the 
trade  fly-off.  Each  test  card  lists  the  related  objective. 

4.2.2  PROOF  OF  CONCEPT  TEST  PLAN  OBJECTIVES 

To  determine  the  economic  feasibility  and  effectiveness  of  delivering  TBMCS 
operator  type  I  training  the  following  four  objectives  was  identified  for  testing  and 
demonstration  purposes: 

•  Objective  1 :  Assess  the  VU  Performance  on  the  DOD  network  and 
computing  infrastructure 

•  Objective  2:  Assess  the  overall  capability/functionality  of  VU  software  to 
insure  compatibility  with  TBMCS  unique  training  solutions 

•  Objective  3:  Assess  the  ease  of  use  of  the  VU  software  by  students  and 
instructors 

•  Objective  4:  Assess  the  task- technology  fit  to  determine  if  the  VU  is  an 
appropriate  technology  for  teaching  TBMCS  tasks. 

4.2.3  TEST  CARDS,  TESTER’S  SURVEY  AND  OBSERVER’S  GUIDE 

During  the  formal  and  informal  tests,  participants  were  requested  to  complete  and 
return  the  Test  Cards  and  a  Usability  and  Technology  Acceptance  Survey.  The 
results  of  both  the  Test  Cards  and  the  Survey  are  documented  in  sections  4.4.4 

An  Observer’s  Guide  was  provided  to  inform  the  test  observers  who  were  not 
participants  with  standards  for  evaluating  the  VU  tests. 


4. 2. 3.1  Test  Cards 

Once  a  student  was  identified  as  a  VU  partieipant,  he  or  she  was  provided  a  set  of  the 
seven  test  cards.  Each  student  was  required  to  complete  and  return  the  Test  Cards  to 
LM  upon  completion  of  the  VU  demonstration.  Test  Cards  #1  and  #2  were  to  be 
completed  prior  to  the  beginning  of  class.  The  remaining  five  Test  Cards  were 
completed  during  class.  Additionally,  students  were  asked  to  fill  in  the  results  on  the 
test  eards  of  a  pretest,  a  posttest,  and  a  eritique  that  was  given  during  elass.  The 
rating  criteria  for  the  Test  Cards  were  Pass  (4),  Some  Problems  (3),  Fail  (2),  and  N/A 
(1).  For  those  answers  with  Some  Problems  or  Fail,  students  were  asked  to  provide 
written  comments  on  the  Test  Card.  The  seven  Test  Cards  are  listed  below. 

•  Test  Card  #1  -  Day  before  class— Student  eompletes  Pre-Class 
Preparation. 

•  Test  Card  #2  -  Day  of  elass— Student  enters  classroom  and  receives  and 
introduction  to  the  elass 

•  Test  Card  #3  -  Student  reeeives  an  explanation  and  demonstration  of 
building  a  mission  shell  or  updating  a  mission’s  status, 

•  Test  Card  #4  -  Supervised  Practiee— Student  attempts  to  complete  first 
exercise  and  reeeives  remediation. 

•  Test  Card  #5  -  Independent  Praetiee — Student  eompletes  the  second 
exercise. 

•  Test  Card  #6  -  Close  Class 

•  Test  Card  #7  -  Student  receives  a  demonstration  and  look-step  practice  of 
updating  a  mission  status  in  ESTAT. 

4. 2. 3. 2  Usability  and  Technology  Acceptance  Survey 

Fike  the  Test  Cards  the  students/observers  were  provided  with  the  Usability  and 
Technology  Acceptance  Survey.  At  the  oompletion  of  class  the  Survey  was  to  be 
returned  to  EM.  The  rating  criteria  for  the  Survey  were;  Strongly  Agree  (4), 
Somewhat  Agree  (3),  Somewhat  Disagree  (2),  and  Strongly  Disagree  (1).  For 
answers  other  than  Strongly  Agree,  students  were  asked  to  provide  written  comments 
on  the  survey.  The  Survey  contained  the  following  seven  sections: 

•  Part  I  -  Tester  Information 

•  Part  II  -  System  Capabilities 

•  Part  III  -  Pre-Class  Preparation 

•  Part  IV  -  Collaboration  Software  Ease  of  Use 

•  Part  V  -  Class  Instruetion 

•  Part  VI  -  Resourees 

•  Part  VII  -  Overall  User  Reaction 

4,3  PERFORMANCE  MONITORING 

The  goal  of  the  performance  monitoring  was  to  capture  objective  measurements  of  key 
performanee  metrics  that  have  been  identified  as  success  criteria  for  the  TBMCS  VU 
system.  In  order  to  monitor  the  performance  of  the  products  several  methods  were  used 
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during  testing  and  demonstrations  as  well  as  methods  used  to  track  on-going  systems 
performance. 

As  part  of  this  effort  it  was  important  to  have  a  process  that  would  monitor  performance 
of  the  software  products  installed  on  the  TBMCS  VU  training  suite.  This  process  would 
provide  the  necessary  data  to  monitor  and  measure  system  performance  and  to  record 
bandwidth.  Mercury  Interactive  was  hired  to  provide  products  and  technical  services  to 
monitor  the  performance  of  the  VU  classroom  environment.  A  test  plan  was  initiated  by 
LM  personnel  and  followed  through  by  Mercury  Interactive  personnel.  Topaz  Prism  and 
Topaz  Managed  Services  (TMS)  were  the  two  products  selected  to  The  Mercury 
Interactive  personnel  first  arrived  at  LM  site  on  22  Sep  03  to  install  and  provide  training 
on  Topaz  Prism  server  and  probe. 

Although  these  products  were  useful  for  our  demonstrations  they  were  not  designed  to 
find  network  or  system  problems  instantaneously.  These  products  would  be  more  useful 
in  an  established  environment  in  an  on-going  basis. 

The  performance  monitoring  results  summary  is  documented  in  sections  4.4.4  and  4.4.5. 
To  further  review  the  results  of  VU  system  performance  refer  to  TBMCS  VU 
Performance  Test  Results  (ref.  5  and  6). 


4.4  CENTRA  DEMONSTRATIONS  AND  RESULTS 


4,4,1  INFORMATION  ABOUT  CENTRA  SOFTWARE  INC 

Centra  software  has  been  in  business  for  more  than  eight  years.  Centra  Software 
provides  enterprise  software  and  serviees  for  real-time  eommunieation,  eollaboration 
and  learning  over  the  Web,  with  self-serviee  meetings,  web  seminars  and  virtual 
elasses.  Centra  produets  are  used  by  more  than  1200  organizations  worldwide  with 
more  than  three  million  users.  Centra  Software  has  support.  Quality  Assuranee,  and 
development  teams  to  provide  any  support  to  eustomers.  Centra  supports  all  of  the 
software  produets  written  and  sold  by  Centra  Software.  To  further  review 
information  on  Centra  refer  to  TBMCS  VU  Trade  Study  Analysis  Centra  (ref.  7) 


4,4,2  DEMONSTRATIONS 

4. 4. 2.1  Demo  Set  #1  -  Informal  Demonstration  (14  October  2003) 

The  first  Government  demonstration  was  eondueted  on  14  Oetober  2003  at  1:00  PM 
Mountain  Time.  Prior  to  the  demonstration  a  one  hour  Pre-test  brief  was  eondueted 
on  12  Oetober  2003  to  provide  an  overview  of  the  VU  demo  purpose  and  to  provide 
speeifie  instruetion  for  aeeessing  the  TBMCS  website  and  the  VU  elassroom.  The 
seheduled  two-hour  demonstration  lasted  for  the  full  two  hours,  and  based  on  the 
results  of  the  Test  Cards  and  Tester’s  Survey,  the  demonstration  was  a  sueeess. 


4. 4. 2. 2  Demo  Set  #2  -  Formal  Demonstration  (23-24  October  2003) 

The  seeond  set  of  Government  demonstrations  were  to  be  eondueted  on  23  and  24 
Oetober  2003.  The  first  demonstration  was  eondueted  at  1:00  PM  Mountain  Time  on 
23  Oet.  The  first  hour  was  moving  along  smoothly.  At  approximately  2:00  PM 
serious  system  problems  oeeurred  and  within  10  minutes  all  22  partieipants  were 
dropped  out  of  the  eourse.  An  attempt  was  made  to  have  all  the  partieipants  re-enter 
the  elass.  Within  10  minutes  again  everyone  was  dropped  out.  The  demonstration 
was  stopped.  Upon  investigating  the  problem.  Centra  teehnieal  personnel  resolved 
that  the  oeeurrenee  was  due  to  not  having  the  most  current  Centra  Service  Pack 
installed  on  the  TBMCS  VU  server.  Because  the  system  was  not  operable,  the  Friday 
event  was  cancelled. 

At  the  time  the  problem  was  identified  a  joint  decision  between  the  ESC  SPO  and  LM 
was  made  to  conduct  three  additional  demonstrations  4-5  December.  The 
additional  demonstrations  could  not  be  conducted  immediately  following  the  problem 
occurrence  because  Vendor  #2  was  scheduled  to  arrive  at  the  LM  site  on  Monday  27 
October.  Testing  and  demos  were  scheduled  for  the  following  four  weeks. 
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4.4.23 Demo  Set  #3  -  Formal  Demonstrations  (4-5  December  2003) 


The  third  set  of  Government  demonstrations  were  eondueted  4-5  Deeember  2003. 
Two  demonstrations  were  eondueted  on  4  Deeember  and  one  demonstration  was 
eondueted  on  5  December.  The  first  demonstration  on  4  December  was  conducted  in 
it  entirety.  The  second  demonstration  on  4  December  had  one  student  from  Osan  AB, 
Korea.  The  student’s  homebuilt  machine  inadvertently  rebooted  two  times.  After 
waiting  for  15  minutes  for  the  student  to  return  class  was  dismissed.  The  third 
demonstration  on  5  December  was  conducted  in  its  entirety. 


4,4,3  TEST  CARD  AND  TESTER’S  SURVEY  RESULTS 

4. 4. 3.1  Test  Card  Results 

Survey  results  illustrate  a  greater  than  92%  positive  response  to  VU  training  overall. 
Six  areas  were  focused  on  within  the  surveys  to  include:  Pre-Class  Preparation, 
Collaboration  Software  Ease  of  Use,  Class  Instruction,  Resources  and  Overall 
Reaction.  In  every  area  surveyed,  strong  agreement  was  found  in  over  75%  of 
respondents  as  to  the  positive  reception  of  the  VU  session.  Almost  20%  of 
respondents  in  every  area  agreed  and  less  than  5%  disagreed  strongly. 


4. 4. 3. 2  Tester  s  Survey  Results 

Survey  results  indicate  a  strong  willingness  to  utilize  VU  training  with  almost  94%  of 
respondents  agreeing  or  strongly  agreeing  to  VU  training.  15  out  of  16  respondents 
would  recommend  VU  training  to  others.  14  out  of  15  respondents  stated  that  VU 
interactions  between  instructor  and  students  closely  emulated  the  interactions  in  a  live 
classroom. 


4. 4. 3. 3 Instructor  Observations  and  Assessment 

Software  offerings  that  perform  the  functions  required  of  VU  are  minimal.  Two 
technologies  were  compared.  Centra  Symposium  and  iLink  Learning  Center  were  the 
two  that  met  all  requirements.  Utilizing  a  comparison  matrix.  Centra  Symposium  was 
chosen.  Symposium  offered  a  great  deal  more  functionality  and  ease  of  use. 


4,4,4  PERFORMANCE  MONITORING  RESULTS 


4.4.4. 1  Summary 

During  the  23  October  TBMCS  VU  demonstration  the  interactive  training  application 
was  aborted  approximately  an  hour  in  to  the  scheduled  two  hour  session.  After 
reviewing  the  system  load,  performance,  and  bandwidth  data  the  problems  appeared 
to  be  unrelated  to  the  sudden  failure. 

During  the  4-5  December  TBMCS  VU  demonstrations  there  appeared  to  be  no 
significant  system  problems.  Some  students  experienced  intermittent  problems. 
These  appeared  to  be  problems  from  the  user  end.  Specific  military  sites  appeared  to 
be  more  prone  to  consistent  system  problems  while  other  military  sites  did  not  have 
systems  problems. 

To  further  review  the  results  of  VU  system  performance  monitoring  refer  to  TBMCS 
VU  Performance  Test  Results  (ref  5  and  6). 


4. 4. 4. 2 Page  Load  Time 

During  the  October  and  December  demonstrations  page  load  time  averaged  sub¬ 
second  response  with  maximum  page  load  time  and  never  exceeding  5  seconds.  This 
held  true  for  all  client  domains  and  connection  speeds.  While  there  appeared  to  be  no 
problems  with  bandwidth,  response  time  or  latency  overall,  there  did  appear  to  be  an 
unusually  high  (>6%)  dropped  packet/retry  rate  on  the  traffic  from/to  the  Langley  Air 
Force  Base  domain.  As  this  is  the  only  client  domain  to  experience  such  a  high  retry 
rate,  it  is  likely  that  it  is  a  local  phenomenon  related  to  Langley  AFB’s  overall 
network  usage  versus  its  capacity. 


4. 4. 4. 3  Audio  Latency  (Delay) 

During  the  October  and  December  demonstrations  audio  latency  for  a  28.8  Kbps 
client  ranged  between  sub-second,  five  seconds,  and  ten  seconds.  Even  when 
intensive  application  sharing  was  being  done,  audio  latency  did  not  exceed  a  10 
second  target.  Audio  latency  for  a  50  Kbps  client  ranged  from  sub-second  to 
approximately  5  seconds.  Audio  latency  for  LAN  speed  connections  was  less  than  5 
seconds. 
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4. 4. 4. 4  Graphic  Latency  (Delay) 

During  the  October  and  December  demonstrations  graphic  latency  for  a  28.8  Kbps 
client  ranged  between  sub-second,  ten  seconds,  and  fifteen  seconds.  Even  when 
intensive  application  sharing  was  being  done,  graphic  latency  did  not  exceed  a  fifteen 
second  target.  Graphic  latency  for  a  50Kbps  client  ranged  from  sub-second  to 
approximately  ten  seconds.  Graphic  latency  for  LAN  speed  connections  was  less 
than  five  seconds. 


4. 4. 4. 5  Bandwidth 

During  the  October  and  December  demonstrations  the  student  client  connection  at 
28.8Kbps  was  the  minimum  connection  speed  measured  during  the  demonstration 
that  met  the  minimum  performance  requirements.  The  typical  operating  range  for 
network  data  transmission  was  between  20Kbps  and  50Kbps.  Higher  spikes  were 
noted  for  clients  on  LAN  connections  (example:  LM)  that  could  handle  a  faster  data 
receipt  and  acknowledgement.  The  connection  at  28.8  Kbps  handled  the  traffic  by 
sustaining  its  maximum  throughput  for  longer  periods  for  each  transmission  from  the 
server  to  the  clients.  Transmission  from  the  server  was  never  sustained  at  a  high 
enough  volume  for  a  long  enough  period  for  the  28.8Kbps  client  to  experience 
latency  greater  than  ten  to  fifteen  seconds. 


4.5  ILINC/NS  SOFTWARE  DEMONSTRATIONS  AND  RESULTS 

4,5.1  ILINC/NS  SOFTWARE  INFORMATION 

EDT  Learning  (iLinc)  has  been  in  business  since  1998.  Their  primary  focus  is  on  e- 
Learning  product  solutions.  As  mentioned  above  NS  Software  presented  to  VU  a 
fully  integrated  product  including  a  ELMS.  EDT  Learning  software  has  been  in 
business  for  more  than  eight  years.  To  further  review  information  on  iLinc/NS 
Software  refer  to  TBMCS  VU  Trade  Study  Analysis  iLinc  (ref.  8) 


4,5.2  DEMONSTRATIONS 

Because  of  the  instability  of  the  software  informal  and  formal  tests  were  not 
completed.  The  continuous  participant  drop  offs  prevented  the  ability  to  conduct  a 
course  in  a  normal  manner. 


4,5.3  TEST  CARD  AND  QUESTIONNAIRE  RESULTS 

Test  Cards  and  Tester’s  Surveys  are  not  available. 


4,5.4  PERFORMANCE  MONITORING  RESULTS 

Although  iLinc/NS  Software  informal  and  formal  demonstrations  were  not 
eonducted,  a  high-level  analysis  was  performed.  On  17  November  an  informal  test 
was  eondueted  to  determine  if  ILine/NS  Software  was  stable  enough  to  eontinue  with 
the  informal  and  formal  Government  demonstrations.  The  test  lasted  no  more  then 
one  hour. 


5  VIRTUAL  UNIVERSITY  CONFIGURATION 


5.1  SYSTEM  ARCHITECHTURE 

The  TBMCS  VU  system  is  made  up  of  the  following  eomponents  (ref  14) 

•  Loekheed  Martin  Network  Firewall 

•  LM  Network  Address  Translation  (NAT)  Router 

•  TBMCS  Training  Web  Server 

•  TBMCS  VU  Web  and  Virtual  Classroom  Server 

•  TBMCS  Lite  Applieation  Suite 

•  TBMCS  Portal  Server 

•  TBMCS  Part  Task  Trainer  (PTT,  provides  simulated  SAA  feeds) 

•  Two  Instruetor  Laptops  with  a  headset  for  eaeh  instruetor 

•  One  Moderator  Laptop  with  headset 

5.2  NETWORK  SECURITY 

As  with  all  DOD  network  eomputer  systems,  information  seeurity  is  required.  The 
TBMCS  VU  system  is  not  eonneeted  to  an  operational  system  and  eontains  no  Real 
World  data  that  would  be  considered  sensitive  or  classified.  LM  has  taken  steps  to  secure 
areas  of  the  actual  TBMCS  system  to  protect  the  database  structures  and  systems  calls. 
The  largest  measure  we  have  taken  is  to  place  the  TBMCS  VU  system  behind  the 
TBMCS  Corporate  Firewall.  This  stateful  inspection  firewall  protects  the  network  from 
malicious  attacks  and  code.  All  access  to  the  VU  components  are  accessed  using  the 
Secure  Socket  Layers  (SSL)  encryptions  for  additional  security.  We  have  also 
implemented  the  requirement  of  logins  and  passwords  into  the  VU  systems  components. 


5.3  INFORMATION  SENSITIVITY  (CONVERSION  TO  SI. 1.3) 

Major  impacts  of  the  conversion  to  TBMCS  Spiral  1.1.3  will  be  as  follows: 
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Hardware: 


•  TBMCS  SI .  1 .3  has  specific  hardware  requirements  needed  to  support  the 
Solaris  8  and  Oracle  9i  architectures,  as  well  as  the  WebLogic  web  server. 

•  More  PC  based  hardware  is  needed  as  the  majority  of  the  applications  are 
either  web-based  or  Microsoft  Windows  GUI-based. 

•  The  VU  server  will  need  to  be  a  multi-processor  PC-based  system  with  a 
minimum  of  2GB  of  RAM,  and  a  minimum  of  100GB  of  drive  space  to 
allow  for  storage  of  multiple  recordings  of  classes. 

Software: 


•  TBMCS  SI .  1 .3  is  Solaris  8  based  which  means  an  entire  rebuild  of  the 
TBMCS  VU  system  will  need  to  be  implemented. 

•  TBMCS  S 1 . 1 .3  is  being  designed  and  developed  to  be  compatible  with 
Windows  Internet  Explorer  (IE)  6  and  Netscape  Navigator  7. 

•  TBMCS  VU  is  based  on  the  Microsoft  Windows  2000  Server  architecture 
using  IIS  5  for  a  web  server. 


6  RETURN  ON  INVESTMENT 


On  the  surface,  the  largest  tangible  savings  for  live  learning  will  accrue  from: 

•  Decreased  travel  expenses  for  students  or  instructors 

•  Decreased  physical  classroom  space 


Increased  productivity  because  instructors  and  students  can  stay  on  task  and  continue  to 
get  part  of  their  day  job  accomplished 

•  Eower  course  distribution  costs 

•  Better  retention  of  information/knowledge  because  the  classroom  material  has 
been  chunked  and  delivered  over  time  as  opposed  to  being  dumped  onto 
students  all  at  once. 

•  A  wider  student  populace  is  trained  because  individuals  may  not  have  the  time 
budget,  or  availability  to  attend  a  traditional  class. 

•  The  best  instructors  can  be  made  available  worldwide  at  the  same  time  to 
support  user  needs. 


7  FINAL  CONCLUSIONS 


The  Proof  of  Concept  classes  held  using  Centra  and  the  results  from  the  Test  Cards  and 
Survey  showed  that  the  virtual  classroom  did  prove  to  be  a  viable  alternative  method  of 
providing  Type  I  Operator  Training  via  the  WWW.  The  ROI  shows  that  after  the  initial 
investment,  VU  can  provide  Type  I  Operator  Training  at  a  significantly  reduced  cost  over 
MTT  costs  for  travel  and  training. 

While  VU  may  not  be  able  to  provide  all  Type  I  Operator  Training  in  all  situations  and 
locations,  it  can  provide  required  training  at  a  reduced  cost  and  an  increased  efficiency  in 
terms  of  response  time.  VU  classes  can  be  scheduled  or  delivered  on  an  as  needed  basis 
with  a  very  short  turn  time. 


8  RECOMMENDATION 

Since  the  second  vendor,  NS  Software,  product  was  not  stable  enough  to  conduct  the 
formal  demos  our  recommendation  is  not  to  determine  which  vendor  has  a  more  viable 
solution  rather  we  have  provided  enough  data  and  results  for  Centra’s  virtual  classroom 
that  our  intent  is  to  recommend  implementing  Phase  II  and  begin  to  convert  courseware 
to  the  VU. 
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Overview 
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Problem  Definition 

Electronic  Systems  Center  (ESC)/ACF  Theater  Battle 
Management  Core  Systems  (TBMCS)  system  program 
office  training  requirements  continued  to  increase  and 
training  budget  continued  to  decrease 

TBMCS  end  users  dispersed  world  wide 

Approximately  50%  of  training  budget  for  Mobile  Training 
Team  (MTT)  travel  costs 

High  ops  tempo  for  user  base  (military) 

Need  for  anytime,  anywhere  training 

Compliance  with  DOD  Advanced  Distributed  Learning 
(ADL)  concept 

Very  little  lessons  learned  /best  practiees  amongst  the  DOD 
community 
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Solution  -  Proof-of-Concept 

Trade  Study 

•  Requirements  committee  opened  action  item  for  TBMCS 
System  Program  Office  to  research  feasibility  of 
implementing  alternative  training  to  MTT. 

•  ESC/ACF  conducted  trade  study  Jun-Dec  03  to  determine 
if  a  Virtual  University  (VU)  is  an  effective  alternative  to 
MTTs.  Key  concepts  included: 

-  Asynchronous/Synchronous  Training 

-  Application  Sharing 

-  Collaborative  Tools 

-  Run  over  NIPRNet/Intemet 
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Key  Concepts  Defined 

•  Synchronous  Training 

-  Instructors  and  students  are  in  the  classroom  real  time 

-  Instructor  and  students  are  geographically  separated  but 
connected  by  computer  network 

-  Emulates  a  live  classroom 

-  Hands-On  Practice  with  Instructor  Over-the  Shoulder 

•  Asynchronous  Training 

-  Recordings  of  Synchronous  classes 

-  Self-paced/web-Based  training  courses 

-  Training  materials 


06/13/05 


5 


Research  Questions 


1 .  Can  joint  training  requirements  be  effectively 
met  via  synchronous/asynchronous  distance 
learning? 

2.  Can  the  current  DOD  infrastructure  support 
synchronous  distance  learning  technologies? 

3.  Is  there  a  cost  savings  for  a  virtual  course  vs. 
MTTs? 
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VU  Proof-of-Concept 
Test  Phases 


•  Phase  I  -  Request  for  Information 

-  Select  Vendor  Candidates 

•  Phase  II  -  Fly-off  Between  Vendor  Finalists* 
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Phase  II 

VU  Fly -off  Testing 

Four  Objectives 

1 .  Assess  the  VU  performance  on  DOD  network  and 
computing  infrastructure 

2.  Assess  the  overall  capability/functionality  of  VU  SW 
to  insure  compatibility  with  TBMCS  unique  training 
solutions 

3.  Assess  the  ease  of  use  of  the  VU  by  students  and 
instructors 

4.  Assess  the  task-technology  fit  to  determine  if  the  VU 
is  an  appropriate  technology  for  teaching  TBMCS 
tasks 
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Methodology 
Evaluation  Types* 


Approach 

Description 

Collection  Method 

Objective-oriented 

Approach 

The  focus  of  this  approach  is  on 
specifying  goals  and  objective  and 
determining  the  extent  to  which 
thev  have  been  attained. 

Type:  Test  Cards 

Who:  Participants,  Observers, 
Instructors 

Participant-oriented 

Approach 

This  is  dependent  upon  the 
involvement  of  the  participants. 
Stakeholders  are  central  in 
determining  the  values,  criteria, 
needs  and  data  for  the  evaluation. 

Tvpe:  Test  Cards/TAM  Survev 

Who:  Participants,  Instructors 

Experience- 
oriented  Approach 

This  approach  is  dependent  upon 
the  direct  application  of 
professional  expertise  to  judge  the 
training  objectives  can  be  met  in  a 
virtual  environment. 

Tvpe:  Observation 

Who:  Subject  Matter  Experts 

Management- 
oriented  Approach 

This  approach  is  used  when  the 
central  concern  is  on  identifying 
and  meeting  the  informational 
needs  of  managerial  decision 
makers. 

Tvpe:  Svstems  Evaluation 

Approach 

Who:  Program  Manager 

*Worthen,  B.,  Sanders,  J.,  Fitzpatrick,  J  (1997).  Program  Evaluation  alternative  Approaches  and  Practical  Guidelines.  Longman  NY. 
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Methodology 

VU  Test  Collection  Instruments 


Test  Cards  -  Derived  from  Test  Plan  Objectives 

-  Individual  test  cards  to  measure:  ease  of  use  for  pre-class  preparation, 
entering/exiting  classroom,  capability  for  student  to  receive 
lecture/demonstration  of  a  learning  objective,  student  attempts  to  complete 
exercises,  independent  practice,  etc. 

Observer  Field  Notes  -  Log 

-  Student  reactions,  voice/data  quality,  etc. 

Technology  Acceptance  Model  Survey*  -  Captured  Qualitative, 

Overall  Impressions 

-  Perceived  usability  &  ease  of  use 

Performance  Monitoring  -  System  Metrics 

-  Topaz  Prism/Mercury  Interactive  tool 


*Technology  Acceptance  Model  Defined  —  http://www.absoluteastronomy.eom/encyclopedia/T/Te/Technology_acceptance_model.htm 
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Methodology 
VU  Test  Activities 

•  Student  logs-in  and  enrolls 

•  Student  takes  pre-test 

•  Student  down-loads  needed  worksheets  and  handouts 

•  Class  in  session* 

-  Instructor  explains  and  demonstrates  TBMCS  operation 

-  Students  practice  using  real  application 

-  Instructor  provides  individual  over-the-shoulder  help  as  needed 

-  Student  takes  post  test  and  completes  critique 


*  system  performance  metrics  collected 


Activities  emulate  a  traditional  classroom  environment 
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Methodology 
VU  Test  Participants 


Organization 

Location 

Number 

of 

T  esters 

Number  of 
Tests  They 
Participated  In 

12AF 

Davis-Monthan 

3 

2 

7AF 

Osan 

1 

1 

8AF 

Barksdale 

2 

2 

9AF 

Shaw 

5 

2 

ACC/DOY 

Langley 

2 

4 

CENTAF 

Al  Udeid 

1 

1 

ESC/ACF 

Hanscom 

6 

7 

NORAD 

CMOC 

3 

4 

PACAF/DOQ 

Hickam 

1 

3 

USAFE 

Ramstein 

3 

1 

USMC 

Hanscom/Hurlburl 

3 

1 

USN 

St.  Juliens  Creek 

2 

3 

Representative  of  target  audience 
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Methodology 
Limitations 

•  Paradigm  Change 

-  Perceptions/attitudes  that  on-line  environments  are  not  as  efficient 
(both  instructors  and  students) 

-  A  new  way  of  teaching/leaming  for  everyone 

-  Determining  non-verbal  feedback  from  students 

•  Bandwidth  at  Selected  DoD  Installations 

•  Program  Level  Security 
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Findings 

Data  Results  -  Test  Cards 


•  Tests 

-  One  Connectivity  Test 

-  Two  Informal  Tests 

-  Four  Formal  Tests 

•  Seven  Test  Cards 

-  Class  Prep 

-  Classroom  Entry 

-  Intro  to  Class 

-  Instructor  Demo 

-  Supervised  Practice 

-  Independent  Practice 

-  Close  Class 

•  Participants 

-  42  total  participants 

-  28  returned  surveys 

-  66%  response  rate 


Total  Responses* 


N/A 

Fail 

Some 

Problems 

Pass 

79 

11 

49 

1011 

Test  Cards  Totals  Summary 


*  frequency  count 
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Findings 

Data  Results  -  TAM  Survey 


Test 

-  Surveys  conducted  simultaneously 
with  the  test  cards  and  were  turned 
in  at  the  conclusion  of  each  test 

Survey  foeus 

-  Perceived  usefulness  -“The  degree 
to  which  a  person  believes  that 
using  a  particular  system  would 
enhance  his  or  her  job 
performance”  (Davis,  1989). 

-  Perceived  ease-of-use  -“The 
degree  to  which  a  person  believes 
that  using  a  particular  system  would 
be  free  from  effort"  (Davis,  1989). 

Participants 

-  51  total  participants 

-  14  returned  surveys 

-  27%  response  rate 


Total  Responses 


Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

2 

4 

21 

67 

TAM  Survey 


Disagree  Agree 

*  frequency  count 


Davis,  F.  D.  (1989).  Perceived  usefulness,  perceived  ease  of  use,  and  user  acceptance  of  information  technology.  MIS  Quarterly,  13(3),  319-340. 
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Findings 

Data  Results  -  Observations 


•  Test 

-  Observations  were 
eondueted 
simultaneously  with 
formal  tests 

•  Observer  Log 

-  Detailed  notes  to 
annotate  observations 

as  they  oeeur. 

•  Foeus  Group 

-  Faeilitate  diseussions 
through  posing  initial 
questions  to  obtain 
reaetions  from  others 
in  the  group. 


“Just  wanted  to  give  you  my  two-cents  worth  of 
impressions  about  Virtual  University.  If  s  great! 

’Nuff  said!?”  Bill  Smith/46  Test  Squadron 

“Great  product.  Good  tool  for  collaborative 

training^ ^  Sheny  Robinson/ESC  Langley  ALB 


“Product  is  easy  to  use  and  captured 
interests/needs  of  testers.  Excellent  choice  for 
on-line  application  sharing” 

Meredith  Briscollino/AOC  Training  Manager 
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Capability  Summary 
(At  End  of  Testing) 

•  VU  can  emulate  live  classroom 

-  Delays  in  audio  and  video  negligible 

-  Instructor  able  to  demonstrate  as  live  classroom 

-  Q&A  interactions  as  effective  as  live  classroom 

-  Students  easily  able  to  operate  actual  TBMCS 
applications 

-  Over- the- Shoulder  Help  as  effective  as  live  classroom 

•  Once  users  participated,  the  attitudes  were  quickly  changed 
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ROI  Expectations 

•  Decreased  travel  expenses  for  students  or  instructors 

•  Decreased  physical  classroom  space 

•  Increased  productivity  because  instructors  can  stay  on  task 
and  continue  to  get  part  of  their  day  job  accomplished 

•  Lower  course  distribution  costs 

•  A  wider  student  populace  is  trained  because  individuals 
may  not  have  the  time  or  budget  to  attend  a  traditional 
class 

•  The  best  instructors  can  be  made  available  worldwide  at 
the  same  time  to  support  user  needs 
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Return  On  Investment  (ROI)  - 
Projected  Estimates 


•  Cost  Per  Student  Training  Hour 

•  MTT  Specific  Costs  (Travel) 

Year  1  Year  2 

-  240  Students  -  $24.68  $26.41 

-  480  Students  -  $12.34  $13.20 

•  VU  Specific  Costs  (Setup/Sustainment) 

Year  1  Year  2 

-  240  Students  -  $25.87  $6.45 

-  480  Students  -  $12.94  $3.23 
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Research  Questions  -  Concluded 


1 .  Can  joint  training  requirements  be  met  via 
synchronous  distance  learning? 

-  Yes 

2.  Can  the  current  DOD  infrastructure  support 
synchronous  distance  learning  technologies? 

-  Yes 

3.  Is  there  a  cost  savings  for  a  virtual  course  vs. 
MTTs? 

-  Yes  (projected) 
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Lessons  Learned 


•  Limit  Class  Size  to  10-15 

•  Provide  “101”  class  separate  from  course 

•  Provide  clearer  registration  and  enrollment 
instructions 

•  Early  sign-in  required 

•  Local  NIPRNet  workstations  security 
lockdowns  problematic 
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Summary 


VU  Proof-of-Concept  trade  study  was  an 
effective  means  to  identify,  assess  and  select  a 
vendor  that  would  best  meet  the  Government 
needs 

Final  vendor  demonstrated  on-line 
synchronous/ asynchronous,  application 
sharing  capability  which  is  beneficial  to  the 
Government  in  solving  training  problems 

VU  product  is  a  viable  solution  as  an 
alternative  to  MTTs 


Training  when 
you  need  it! 


Virtual  University 
delivers  training 
when  you  need  it, 
wherever  you  are. 


Virtual  Universitj'  is  being 
pioneered  through  the  Theater 
Battle  Management  Core 
Systems  (TBMCS)  program,  the 
system  that  plans  and  manages 
all  joint  and  coalition  air 
operations. 
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Back 
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Network  Architecture 


TBMCS  Training  Virtual 
University  Configuration 

^  ^  TBMCS  Student  Clients 
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VU  Test  Suite  Hardware 

•  Windows  PC  with  Headset  or  Microphone  and  Speakers 

-  2  Instructor  Workstations 

-  1  Moderator  Workstation 

-  1  Support  (Help  Desk)  Workstation 

-  Multiple  Student  Workstations 

•  TBMCS  Lite  Application  Suite  (Unclassified) 

-  ORASVR  (Fictional  Dataset) 

-  SYBSVR  (Fictional  Dataset) 

-  WebLogic  (Enhanced  Web  Applications) 
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VU  Test  Suite  Software 

•  Software  Plugin  (Automatically  downloaded) 

-  Collaboration  Software  and  Web  Browser 

•  Certificate  (Verisign)  for  Secure  Connection 

•  Asynchronous  Downloads  of  Course  Materials 

•  Communicates  Via  the  Internet  and  NIPRNET 

-  Viable  Voice  Over  IP  (VOIP)  Performance  Via 
Connections  As  Low  As  28Kbps  Dial-Up 

-  All  Communication  Transferred  Via  HTTP  and  HTTPS 
Ports  80,  and  443 
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Phase  I  -  Request  for  Information 
Select  Vendor  Candidates 


RFI  sent  to  8  vendors  -  4  responded 

-  ATC  -  Intranet  U 

-  Centra 

-  Click2Leam 

-  LeamLinc  (NS  Software) 

VU  Team  researched  all  4  vendors  for  current  customer 
satisfaction 

-  Centra  and  Leamlinc  customers  -  Highly  Satisfied 

Based  on  a  Weighted  Rating  Scale  two  vendors  -  Centra  and 
Leamlinc  products  -  received  the  highest  score 

-  Centra  and  Leamlinc  met  all  of  the  requirements 

Proceed  with  Phase  II-  VU  Testing 
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Test  Card 
Example 


TAM  Example 


Example  Test  Instruments 


Test  Card  #2,  Classroom  Entry 

N/A 

Fail 

Some 

Problems 

Pass 

student  enters  classroom  and  receives  an 
introduction  to  the  class 

1 

You  access  TBMCS  Website 

2 

You  log  into  the  TBMCS-Centra  VU  Homepage 

3 

You  enter  the  VU  classroom  within  30  minutes  of  class  start  time 

4 

Voice  communication  established  between  tester  (you)  and  instructor 

5 

You  observe  introduction  slides 

6 

You  can  easily  identify  each  slide 

7 

Instructor  can  easily  display  an  earlier  slide  in  response  to  a  student's 
question 

8 

Screen  and  audio  stay  in  sync 

7.1 

7.2 

7.3 

7.4 

7.5 

7.6 


Technology  Acceptance  Model  Survey 


I  would  encourage  my  unit  to  use  the  VU  for  TBMCS 
training. 

If  I’m  unable  to  get  live  TBMCS  training,  I  will  make  time  to 
attend  VU  TBMCS  classes. 

The  VU  technology  provides  a  suitable  alternative  to  live 
training. 

I  would  recommend  the  VU  system  to  others  for  training. 

If  I  couldn’t  have  a  live  instructor,  I  would  want  to  attend  the 
VU. 

The  VU  interactions  between  instructor  and  students  closely 
emulated  the  interactions  in  a  live  classroom. 


Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 
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Methodology 
Test  Schedule 


Event 

Type  Test 

Dates 

Vendor  1 

Informal 

Oct  03 

Vendor  1 

Formal/User  Community 

Oct  03 

Vendor  2 

Informal 

Nov  03 

Vendor  2 

Formal/User  Community 

Nov  03 

Vendor  1 

Informal 

Nov  03 

Vendor  1 

Formal/User  Community* 

Dec  04 

Vendor  1 

Formal/User  Community* 

Dec  04 

Vendor  1 

Formal/User  Community* 

Dec  04 

*Multiple  time  zones 
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Test  Results 


Event 

Type  Test 

Dates 

Results# 

Vendor  1 

Informal 

14  Oct  03 

Satisfactory 

Vendor  1 

Formal/User  Community 

23  Oct  03 

Unsatisfactory* 

Vendor  2 

Informal 

17  Nov  03 

Unsatisfactory 

Vendor  2 

Comparative  Connection 

17  Nov  03 

Elimination  of  Vendor  #2 

Test* 

Satisfactory  (2  hr 

Vendor  1 

Stress  Test* 

20  Nov  03 

connection) 

Vendor  2 

Formal/User  Community 

4  Dec  04 

Satisfactory 

Vendor  2 

Formal/User  Community 

4  Dec  04 

Satisfactory 

Vendor  2 

Formal/User  Community 

5  Dec  04 

Satisfactory 

*  23  Oct  test  resulted  in  an  unknown  loss  in  conneetivity.  Performanee  metrics  did  not  show  loss  in  bandwidth. .  .required 
additional  connection/stress  testing  to  determine  eause 

#  Total  results  included  system  eapabilities,  overall  user  reaction,  and  the  notation  of  any  significant  problems 

eneountered  • 

06/13/05  Not  sure  if  I  like  this  30 

slide 


